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ABSTRACT 

With the onset of Web 2.0 and 3.0 - the social and semantic webs - a next wave for integration of educational 
technology into the classroom is occurring. The aim of this paper is to show how some teachers are increasingly bringing 
collaboration and shared meaning-making through technology environments into learning environments (Evergreen 
Education Group, 2014). Purpose is to show a case study of how teachers include student-to-student online 
collaboration in their Technology Integration Practices (TIP), and how some research projects are examining useful 
methodologies for incorporating evaluation, assessment and reflection of the approaches (Wilson et at., 2012 ; Wilson, 
Scalise, & Gochyyev, 2014). Results from the use of TIP collaborative math/science notebooks in the Assessment and 
Teaching of 21 s ' Century Skills (ATC21S) project are presented. Recommendations are to help teachers answer key 
guestions about how to assess and evaluate collaborative work online, and how to employ such techniques in the 
classroom. 
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INTRODUCTION 

For many teachers, the idea of teaching 21 st century 
standards such as digital collaboration is challenging 
(Partnership for 21 st Century Skills & American Association of 
Colleges of Teacher Education, 2010; Schrum & Levin, 
2014). Teachers ask how they should go about helping 
students build these skills, and wonder what a successful 
performance looks like in a collaborative digital space. 
They want to know how such skills can be effectively 
assessed, and whether and how students should be 
expected to improve over time. Instructors have a lot of 
experience recognizing more traditional work products in 
the classroom, but sometimes don't know if they can 
effectively recognize increasing student proficiency in an 
area such as digital collaboration. They haven't seen many 
examples. 

In this study, a single case study of collaborative work 
products from the Assessment and Teaching of 21 st Century 


Skills (ATC21S) project was examined. The case study uses 
one type of collaborative team notebook activity in a 
math/science learning environment. 

ATC21S is an alliance launched by Cisco, Intel, and 
Microsoft and including government, schools, and 
university partners across numerous countries. Goals 
include to encourage the development of 21 st century 
learners and to enhance the skills of the workforce of 
tomorrow (Griffin, McGaw, &Care, 2012). 

The ATC21S project describes today's International and 
National Educational Standards as primarily measuring 
performance in core subjects like Math, Science and 
Reading. Although innovative inclusion of 21 st century skills 
in some schools exists, limited mainstream implementation 
means that many teachers are having to develop their 
own approaches to both teaching and assessment. Similar 
to the Partnership for 21 st Century Skills (2009 & 2010), 
ATC21S describes that, there are few curriculum and 
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assessment tools in place to help schools succeed at 
teaching critical 21 st century skills (Griffin et al., 2012; 
Partnership for 21 st Century Skills, 2009; Partnership for 21 st 
Century Skills & American Association of Colleges of 
Teacher Education, 2010), 

For the case study, a digital collaboration in on ATC21S 
Science and Mathematics activity is examined (Scalise, 
2013; Wilson & Scalise, 2012), The case study introduces 
ways of viewing and interpreting collaborative 
performances online. It discusses some key attributes of 
successful digital collaboration that are easy for teachers 
to recognize in classroom work products. 

Need and Rationale for the Study: Digital Collaboration as 
a 21 st Century Skill 

One key topic that teachers ponder in digital collaboration 
is how to effectively evaluate collaborative work in an 
online setting (McFarlane, 2003). They often feel they are 
good at evaluating work products in their subject matter 
areas, for instance they can "grade" and provide feedback 
for language, math or science competencies in a given 
assignment. But what factors might they tap as indicators of 
growing student proficiency (Wilson et al., 2012) in 
collaborative online digital literacy more generally? Without 
some indicators, it can be difficult for teachers to gauge 
how they are helping students improve in this type of 
Educational Practice. 

Digital literacy as a domain encompasses a wide range of 
subtopics, including learning in networks, information 


3. Learningto learn. 
Metacognition 


literacy, digital competence and technological 
awareness, all of which contribute to learning to learn 
through the development of enabling skills (Wilson & 
Scalise, 2012). For theATC21S project, sets of 21 st century 
skills were identified based on an analysis of twelve relevant 
frameworks drawn from a number of countries and 
international organizations around the world (Binkley et al., 
2012). These included the OECD and countries in Europe, 
North America, Asia, and Australia. Called the "KSAVE" 
framework, the components encompass not only skills, but 
as the acronym implies Knowledge (K), Skills (S), Attitudes 
(A), Values (V), and Ethics (E). KSAVE organizes the ten 
components into four conceptual groupings, and the 
components have been redrawn into a new diagram as 
shown in Figure 1. 

As shown in Figure 1, KSAVE introduces four overall 
groupings for the 21 st century domain: Ways of Thinking, 
Ways of Working, Tools for Working, and Living in the World . 
KSAVE suggests that, the first Ways of Thinking grouping, 
which includes creativity, problem solving and 
metacognition, represents a push forward in the 
conceptualization of thinking. Its components contribute to 
higher order thinking skills, and require cognitive processes 
such as recall and inference, but are not always a current 
focus of formal schooling processes. To incorporate more 
focus in schools requires appreciation of how students 
employ focus and reflection such as in problem solving but 
also indicates how students must be able to take on 
multiple perspectives, value transfer, and apply generative 


10. Personal & social 
responsibility - 
including cultural 
awareness and 
competence 






Figure 1. Components of ATC21S “KSAVE” Framework of 21 st Century Skills, Encompassing Knowledge, Skills, Attitudes, Values and Ethics 
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association, such as are evident in creativity and 
innovation, 

The other groupings in KSAVE have similar group sets within 
the 21 st century domain. For instance, the second 
category, Ways of Working, groups communication and 
collaboration skills together as essential components of 21 st 
century work requirements. Such team contributions, 
knowledge sharing, and dissemination are increasingly a 
top focus of how we work. The third category, Tools for 
Working, includes information literacy and digital literacy in 
the knowledge economy. Finally, Living in the World focuses 
on the broader perspectives of personal, social, local, and 
global citizenship responsibilities and skills. 

For the ATC21S case study, here the focus is on a digital 
literacy portion of Tools for Working, called learning in 
networks, the digital literacy domain for ATC21S is 
comprised of four strands of a learning progression, or set of 
progressively more proficient levels. The four strands are: 

• Functioning as a consumer in networks; 

• Functioning as a producer in networks; 

• Participating in the development of social capital 
through networks; 

• Participating in intellectual capital (i.e., collective 
intelligence) in networks. 

Given the broad reach of these four strands, planning for 
Technology Integration Practices (TIP) that support 
collaboration is not the domain of any one subject matter 
area. Rather examples of effective technology integration 
strategies can be found across many content areas. These 
include language arts/foreign languages, 
mathematics/science, social studies, art/music, physical 
education/health, and special education. 

Thus one way of approaching school use of digital literacy 
is to consider it a practice, or way of working across domain 
but through new tools (Scalise, 2014). Friedman (2007) 
describes such practices as a major shift toward 
technology that Educators need to address. He discusses 
how it may be a counter-production to ask students to 
power down or give up their social media when they enter 
the school doors. Rather, students should actively engage 
in digital literacy practices in formal learning, including 


using the tools, networks and bodies of expertise available 
to students virtually (Scalise, 2013). This both underscores 
developing ICT knowledge and skills as an important 
practice in schools, and allows educators to teach and 
model appropriate use while supporting subject matter 
learning. 

Applying the Concept of TIP to Digital Collaboration in the 
Classroom 

When considering the relationship of the KSAVE learning in 
networks objectives to actual classrooms, it is important to 
consider the technology integration that is done by 
educators in schools (Roblyer, 2004, 2006). Technology 
integration for education is often defined simply as using 
technology as a tool for teaching and learning (Barron, 
Kemker, Harmes, & Kalaydjian, 2003). Scholars in recent 
years have described ICT literacy as often best achieved 
through integrated learning with technology rather than 
stand-alone (Ridgway & McCusker, 2003; Somekh & 
Movers, 2003). In integration, technology use is embedded 
in subject matter areas, authentic tasks, real-world 
problems or other applications. By contrast when taught in 
stand-alone instruction, the focus can be more solely on 
learning a tool rather than its application. Kozma, Jewitt 
and others have called for rethinking both what digital 
literacy calls for, and how technology can better contribute 
to its teaching and assessment (Kozma, 2003; Quellmalz & 
Kozma, 2003). 

For technology integration planning in the classroom, best 
practices include understanding how, when, and why 
technology can be infused into education to improve 
learning outcomes. Poor technology integration planning 
can center on including too much technology in the 
classroom as well as too little. For instance, using 
technology for its own sake rather than strategically to 
support specific learning outcomes can be a problem in 
some schools. Scaffolded hands-on experiences with 
technology and modeling of technology between 
teachers helps to bring about more effective technology 
integration planning. 

Robyler describes technology integration planning or TIP for 
teachers as encompassing five phases: 

• Phase 1: Determine the relative advantage of using 
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technology. 

• Phase 2: Decide on objectives and assessments. 

• Phase 3: Design technology integration strategies. 

• Phase 4: Prepare the instructional environment to 
support successful use of technology 

• Phase 5: Evaluate and revise the integration strategies 
to improve the experience and the learning outcomes 
for students. 

Use of Digital Collaboration in the Classroom 

Teachers use digital collaboration in the classroom today in 
many different ways. One teacher may ask her students to 
compose on Google docs, employing an online 
document in which all of the authors can share access to 
the document so that they can write simultaneously 
(Wessling, 2012). Students grouped into teams of three or 
four work on individual laptops. They sit together in table 
groups. They can speak together face-to-face and discuss 
the developing shared writing in person, but they also 
compose over the computer together for both draft and 
final documents. The online environment affords a variety 
of virtual tools such as for annotation, tagging, and 
providing in-line comments, as well as for tracking prior 
drafts and the contribution of each student. 

Teachers describe that students need to manage these 
skills in productive ways in their later lives. This will involve 
collaboration skills such as turn-taking, affirmation, 
constructive critique, and etiquette, Students can share the 
feedback and ponder solutions together. Teachers can 
see through the online products which students are 
contributing and how. 

A Case Study: Digital Collaboration in Arctic Trek 
Math/Science Notebooks 

One potential mechanism for the assessment of student 
ability in the learning network aspect of ICT literacy is to 
model assessment practice through a set of exemplary 
classroom materials. An ATC21S module that has been 
developed for this is called the "Arctic Trek" scenario and is 
based on the Go North/Polar Husky information website 
(www.polarhusky.com), a project originally of the University 
of Minnesota. The Go North website is an online adventure 
learning project based around arctic environmental 



Figure 2. Opening screen of the ATC21S Arctic Trek Scenario, 
a Collaborative Digital Literacy task in a Science and 
Mathematics Content Area 

expeditions, The website is a learning hub with a broad 
range of information and many different mechanisms to 
support networking with students, teachers and experts. 

The Arctic Trek scenario, with an opening screen shown in 
Figure 2, views social networks through ICT as an 
aggregation of different tools, resources and people that 
together build community in areas of interest. Through a 
series of screens, the tour through the site for the ATC21S 
demonstration scenario is conceived as a "collaboration 
contest," or virtual treasure hunt. In this task, students in small 
teams ponder tools and approaches to unravel clues 
through the Go North site, via touring scientific and 
mathematics expeditions of actual scientists, The items 
from the Arctic Trek scenario are the examples of activities 
(Wilson &Scalise, 2014). 

Scope of Study 

A digital collaborative notebook was employed in different 
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Figure 3. Table showing attributes achieved for the Nine Case 
Study Notebooks Sampled. Teachers can use simple data 
displays like this as Rubrics for Student Teams to self- 
evaluate their Collaboration Efforts 
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Figure 4. Graph displaying Sum Total of Attributes achieved for each 
of the Case Study Notebooks Sampled. Visualizations such as these 
are easy ways to see differences in student work products, and 
can serve as useful Resources for Teachers working on evaluating 
their Student Work during Digital Collaboration Efforts 

trials of the Arctic Trek task in four countries. Nine team 
notebooks are examined in this case study as shown in 
Figures 3 and 4. The Arctic Trek task includes the use of a 
collaborative math/science lab "notebook" shared by 
teams of four students in the classroom. The notebook 
focuses especially on the developing and sustaining 
Intellectual Capital through Networks (ICN) portion of the 
ATC21S KSAVE framework. As a work product, the shared 
notebook is similar to other collaborative online work 
products described for the use in the classroom by 
teachers. The notebook allows groups to "construct" a 
collaboration online. Note that, no face-to-face 
collaboration opportunities are made available in the 
Arctic Trek scenario, so the notebook and its associated 
tools form the full record of the collaborative work product. 
Thus a major point of the Arctic Trek notebook example was 
to yield generative information on the fourth or ICN strand of 
the ICT literacy framework - what do the nine teams of 
students do when faced with opportunities to employ 
collective intelligence? 

As either a Microsoft One Note document or a 
collaborative Google document online, student teams 
were provided with a link in the Arctic Trek activity and a 
"secret code" to login into their shared document online 
with their assigned team members. They then had an 
"open canvas" or mostly blank collaborative work space 
with some associated tools such as a chat box to use for 
their collaboration and communication. 

Containing just a few lines of instruction, the notebook was 


intentionally left as an unstructured, simple device through 
which students could collaborate and share work across 
theirteom. The unstructured nature of the approach meant 
that each team could employ the tool in the manner that 
they thought best and student work across the team could 
be evaluated for attributes capturing how effectively the 
team itself decided to employ their opportunity to 
collaborate. Without substantial layers of scaffolding or 
support telling the students how they should go about the 
collaboration, the teams were therefore left to draw on their 
own working knowledge to gather the team together and 
assist in solving the activities of the task. 

To develop attributes to help interpret the collaborative 
performances seen, a set of Arctic Trek notebooks 
representing a purposive range of performance were 
sampled from the ATC21S project. They were then reviewed 
by methods of integrative research to gather a set of core 
ideas for patterns seen in the student work. These 
represented both some common ground and some 
divergent characteristics by which teachers could 
recognize important attributes of the ICN performance in 
the collaborative artifact. The identification of such 
attributes can help to build a bridge between using such 
approaches in the classroom and evaluating how well 
students are learning to collaborate and employ 
intellectual capital effectively. 

As shown in Figure 3, twelve attributes were identified that 
ranged at initial levels from accessing the digital tool being 
used for collaboration, making attempts to identify team 
members, posing initial questions, and sharing simple 
answers. At more intermediate levels, students are also 
working on at least some types of role allocation, planning 
strategies and shared thinking for the collaborative process 
to the highest level performances participating in effective 
evidence sharing, systematic execution, flexible 
adjustment and analysis during the activities, and 
attempting to come to a shared understanding on tasks 
across the team. 

The twelve attributes identified were analyzed by subject 
matter experts and then sorted in a cognitive diagnostic 
technique of ordering from "easier" attributes to "harder", 
based on sample numbers in which the attributes 
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appeared, with ranking of "ties" by subject matter experts. 
Figure 3 shows the sorting of attributes in the columns, and 
the sorting of notebook's performances from low to high in 
the rows. A zero in the table recorded that, the attribute was 
not present in the notebook; a one recorded that, the 
attribute was present. The fairly regular pattern on the 
diagonal of zeros and ones in the table display indicates 
that, teams with lower performing notebooks tended to 
consistently display the same few attributes while higher 
performing teams compiled both more attributes, and 
higher level attributes on the ICN strand. Figure 4 shows sum 
scores across the attributes by notebook sample. 

Limitations of the Study 

Of course the small sample size of a case study allows 
generative work to take place on new constructs, but larger 
samples and additional methodologies are needed to 
confirm and more broadly generalize interpretations in the 
21 st century skill domains (Wilson et a!., 2012). This is a 
limitation of the case study method. For those interested in 
generalizing into the case study observations from their 
own contexts (Scalise, 2010), the case study format 
enables the goal of showing teachers some potentially 
effective paths to evaluating collaborative tasks in their 
own classrooms. Teachers often have small local samples 
from which they can draw a representative range of work 
like the ATC21S nine notebook set. Similar to a digital case 
study, teachers can then use their usual review of student 
work and their TIP Phase 5 approaches to evaluate the 
patterns in the results. By identifying a small set of helpful 
attributes that show high utility, they have mechanisms to 
guide their own work as well as allow for feedback and 
guidance to students that can be used for both current and 
future evaluations, 

Results Serve as "Anchor” Work Products for Digital 
Collaboration 

Initial proficiency shown by student teams in the notebook 
case study sample indicated, as described in Figure 3, the 
ability to access the collaborative space, begin to identify 
team members, and attempt to pose at least some initial 
questions and answers to the team. Of the nine notebooks 
sampled that represented a range of performance, all 
teams were able to progress this far in their shared efforts. 


These indicators are at the ICN 1 level of the framework. 

Two teams were additionally able to extend to a mid-range 
of collaboration that involved establishing at least some 
partial roles or turn taking in the collaborations, and 
employing the ability to share not only their initial questions 
and answers, but some of the evidence and evaluation 
that the team collected or completed in their information 
for aging during the task. These indicators are at the ICN2 
level of the framework. 

Three teams at the higher performing range of proficiency 
seen on the 12 attributes of Figure 3 show more systematic 
and complete progress in employing intellectual capital 
effectively. Their role planning efforts were more thorough 
and showed evidence of being carried out as agreed 
upon including making adjustments during the task at 
times. The teams independently developed ways to 
systematically identify contributions from different team 
members, and engaged in evidence reconciliation. This 
included identifying signal versus noise in the information, 
interrogating data for meaning, and sharing aspects of 
their mental models during their visualizations of scientific 
data. 

As illustrative examples, three of the notebooks in the case 
study sample will be used to show how teachers can 
explore some degree of "bench marking" for digital 
collaboration work products. The three notebooks again 
represent a range of the performance shown in Figures 3 
and 4, with one selected from each of ICN 1 to ICN3. 

Example 1 in Figure 5(a) is an excerpt from Notebook ID 4. 
This notebook is at the lower end of the collaborative range, 
but still showed the performance of value and skills to be 
reinforced in student digital collaboration. Teachers can 
help such students by pointing out what they are already 
doing effectively as well as continuing to work with them to 
build additional understandings. Achieving four of the 
twelve attributes listed in Figure 3, the team's notebook 
shows proficiencies at the first level, or ICN1 of the ATC21S 
digital literacy "intellectual capital" strand, Students 
correctly access the collaborative tool, make simple efforts 
to identify their team members ("who is this?"; "yes it is 
me!!"), and begin to pose some initial, but relevant 
questions and answers to each other ("you go which 
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is it2??? 

i saw black and grey... ya maybe but i thought also got white? 
yup 

u go which website??? 

i go to polar bear populations because the hint asked me 2. 

i think is 3 cos' in one of the clue they also say which on the map got white polar bear... 
i will click 3. now u post ur ans. so i can copy for what they ask .but we must explain.... i try my 
best ok? 
i oso 

i havent go to tat Question yet.yupppppy 
i keep pressing wrong thing 

orh. 

come on give me a answer 

my answer is 3. Then i will do my research to accompany my answer, 
who is this? 


naila and verretta. r u esan? 
we r doing some research- 
yes it is me!! 

This notebook is at the lower end of the collaborative range, but still showed the performances of value to be reinforced in student digital collaboration, such as 
accessing a collaborative tool, identifying team members, and posing relevant questions and answers. 

Figure 5(a). An Excerpt from Notebook ID 4 


website???", "i will click 3. now u post ur ans. so I can copy 
for what they ask."). 

Figures 5 (a)-(c) are the examples of excerpts from three 
different Arctic Trek notebooks which illustrate different 
ranges of performance seen in the team activities, and 
how teachers can recognize differences in digital 
collaboration performance. 

An Example in Discussing the Use of Colors in 
Representations 

In one part of the task, students are expected to find colors 
that are used to describe the bear population in an online 
table. Requiring both identifying signal versus noise in 
information and interrogating data for meaning, a fully 
successful performance on the "color" task can be 
mapped into the ICN3 level ("Proficient builder") of the ICN 
strand (Wilson & Scalise, 2014), but partial credit is also 
possible. On the color question for the team using 
Notebook 4, the students do not review the correct 
representation, so they are far off in their color 
interpretation. Yet they still show evidence of attempting to 
share their thinking at a level appropriate to ICN1. They post 
simple questions about the color chart they believe that 
they have correctly selected, and they puzzle together 


about some initial answers. However, their postings include 
little or no evidence to support their thinking, and the team 
members do not evaluate shared results or attempt 
strategic thinking across the group to resolve discrepancies 
("i saw black and grey... ya maybe but I thought also got 
white"; no additional commentary following on this topic 
following the discrepant comment). 

By the end of the task, the team shows some basic success 
in accessing and using a digital tool for collaboration, 
establishes the beginning of an ability to "tag" the identity 
of a fellow team member, and is successful in posting 
some relevant questions and answers digitally. Team 
members also show some degree of frustration in their 
attempts to collaborate, for instance answers are posted 
without explanation or evidence: "but we must explain," 
one student tries to correct the group, "i try my best ok?" 
another student responds. Another student reverses the 
temporal order of answer and evidence supplied, telling 
the team, "my answer is 3. Then i will do my research to 
accompany my answer." 

In Example 2 in Figure 5(b), from Notebook ID 9, students 
achieve six out of the twelve attributes in Figure 3. The team 
here is emerging into behaviors of the next level of the 
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Hello? O.o We need to decide who will do what task, 
ok do you want to decide 

Can I pick what I want to do? 

yes we have to make sure we dont end up with the same task 

deose anyone no what we are suppose to be doing, because i would like to do the coloring task 
ay ay hold on a sec. 

i chose four because the graph showed four colors to show the living conditions. 

They use red for declining populations. They use orange for reduced populations They use 
green for not reduced populations. They use light green for stable populations. Then they use 
yellow for moderate. -Jenia 


no it’s five there’s a green light green orange yellow red 

This is an excerpt from a notebook where students achieved six out of the twelve attributes in Figure 4 attributes. 
The team here is entering into the second level of intellectual capital strand of the framework, ICN2. The team shows 
emerging knowledge about the mechanics of collecting and assembly data together in a digital collaboration, and 
of knowing when to draw on collective intelligence. They also begin to acknowledge multiple perspectives by sharing 
some evidence and not just answers across the team during their information foraging. 

Figure 5(b). An Excerpt from Notebook ID 9 


intellectual capital strand of the framework, ICN2. The 
team shows somewhat more knowledge than Team 4 
described above about the mechanics of collecting and 
assembly data together in a digital collaboration, and of 
knowing when to draw on collective intelligence. These are 
some attributes of ICN2. For instance, they make attempts 
at some role planning, which Team 4 did not do ("We need 
to decide who will do what task," one student says, "ok do 
you want to decide," a teammate replies. Other data 
collection planning comments include, "Can I pick what I 
want to do?" and "i would like to do the coloring task."). 

However the team members never systematically agree 
on and execute the planning decisions as a whole. So while 
this team is moving beyond ICN1 and into ICN2, they are 
just beginning to master the skill set of more strategic 
collaboration efforts. These students also begin to 
acknowledge multiple perspectives by sharing evidence 
and not just answers across the team during their 
information foraging. On the color task, one student 
reports, "i chose four because the graph showed four 
colors to show the living conditions." Another student 
disagrees and advances a claim with concrete details as 
her evidence for five colors: "They use red for declining 


populations. They use orange for reduced populations. 
They use green for not reduced populations. They use light 
green for stable populations. Then they use yellow for 
moderate." A third student on the team checks this answer, 
and then confirms the five-color evidence statement. The 
team continues on to have a lengthy discussion about 
other questions and activities in the task. 

In Example 3 in Figure 5(c), which is from Notebook ID 6, 
students achieve nearly all - 11.5 - of the 12 attributes 
identified in Figure 3. Team behaviors show not only ICN2 
traits, but a substantial degree of systematic effort 
associated with the next level of the framework, ICN3. In 
their responses to the "color" question, for instance, the 
team members identify signal versus noise in information as 
they come up with the answer of not only five colors but six, 
which codes for missing data in the task as a source of 
discrepancy for advanced teams to ponder. They record 
their consideration of whether white could be considered a 
color used to represent "data deficiency" or in other words 
the intended missing data, along with recording the five 
color codes for the bear populations actually identified in 
the color chart. 

In other answers from the team, not shown in the Figure 5(c) 
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Roles 

(1) David: Captain, Decoder 

(2) Stephenie:Work on clue 1,2,3 

(3) Xinyi: Work on clue 4,5,6 

(4) Amanda: Scout, Recorder 

Clues 
Clue 1: 

Where the white bear lives. Where on the map do polar bears live who do NOT belong to any 
country? 

Polar bears live in North Pole 


Clue 2: 
Arctic Fox 


Clue 3: 

Answers of person: 

(1 )There are 5 colours of red, orange, light green, dark green and yellow with one extra colour 
white to represent data deficiency 

(2) 5 

(3) 5 

(4) 5 

This is an excerpt from a notebook where students achieve nearly all - 11.5 - of the 12 attributes identified in Figure 4. 

Team behaviors show not only ICN2 traits, but a substantial degree of systematic effort associated with the next 
level of the framework, ICN3. See the text description for additional contents of the notebook, the team members 
establish and employ clear role assignments, identify signal versus noise in information, interrogate data for 
meaning, and share aspects of their mental models during their visualizations of the scientific data. 


Figure 5 (c). An Excerpt from Notebook ID 6 


excerpt since the notebook is extensive, the students from 
this team interrogate data for meaning, They explore how 
the scientific expedition data shows the species under 
consideration is "on the general decrease" and "facing 
high risks of being endangered" but share their thinking to 
introduce caveats. Team members describe areas they 
have discovered in the reporting from the scientific 
expedition that are not consistent with the overall trend, like 
"the Gulf of Boothia, Southern Beaufort Sea and M'Clintock 
Channel" where "risks of future decline are low." 

The students on this team also show evidence of sharing 
aspects of their mental models together. They describe 
their visualizations of data. Todothis, they effectively fit data 
on bear population graphs with digital tools online and 
describe how they "control the gradient/frequency" of the 
display and "the y-intercept" so that they can come up 
with, record, and share with each other the best fitting 
curve for the scientific data, 

However, even for this team, following the reporting of 
discrepant opinions, it is unclear whether the students do 
always consider whether they need to reframe their thinking 


afterwards. The group does not appear to reach a 
consensus or otherwise sum up the conclusions of the 
team as a whole. As they move between questions, they 
sometimes simply report a set of different answers. This 
could be due to having insufficient time in the 45 minute 
task to reconcile answers. However, teams who are able to 
reconcile spontaneously without prompting or scaffolding 
might be considered to be at a higher level of team 
situational awareness and more advanced in their ability to 
negotiate meaning and establish shared understanding 
during digital collaboration. 

Especially in looking ahead to work settings where digital 
collaboration might be an important skill, the expectation is 
often to establish a shared product, such as a common 
recommendation, a report or presentation that synthesizes 
the knowledge of the group. Going beyond what the 
students exemplify in ICN3, this might be an important 
aspect to add to ICN4, as a distinguishing characteristic of 
the top level of the framework. 

Recommendations and Conclusion 

For many teachers, the idea of teaching 21 st century skills 
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such as digital collaboration is both exciting and 
challenging. Teachers want to know how to help students 
build such skills, which many believe will be important in the 
future work and lives of their students. But what does a 
successful performance in digital collaboration look like, 
teachers often ask, and how should they gauge and 
support learning? 

The case study notebooks here reveal a range of 
performance during student collaboration with technology 
can be identified by a set of twelve attributes in digital 
collaboration aligned with ICN levels 1-3 of the ATC21S 
framework. Anchor examples are illustrated in three 
examples. 

Recommendations include that, by examining the 
attributes of digital collaboration and using simple rubrics, 
displays and tabulations of results such as shown here, 
teachers should learn to readily recognize traits for building 
more successful digital collaboration in their student's work. 
Opportunities for learning in social networks supports the 
student experience and enhances growing capabilities for 
effective collaboration. Furthermore, as teachers 
embrace digital tools and collaboration in their 
classrooms, it is recommended that they add to initial 
approaches and customize for the contexts in which they 
educate. 

Teachers who increasingly want students to work together in 
digital environments need to provide avenues for success. 
This includes evidence and feedback on what successful! 
collaboration is expected to look like and to produce. 
Increasingly as the digital age advances, teachers want 
rigorous learning contexts that also employ digital tools and 
learning in social networks for their students. This is 
recommended to contribute both to academic success 
and social/emotional learning, as well as 21 st century skills 
important to students for college and career readiness. 
However, use of assessment and evaluation rubrics for 
these new products in the classroom is recommended. 
These are needed to support teacher understanding and 
student understanding. Teachers need to know what to look 
for and what to teach toward for new digital collaborative 
and socially shared learning activities. Students need to 
know how effectively learning takes place in new learning 


environments and academic situations. 

By mastering productive collaboration in school projects, 
students can gain skills that are valuable for a lifetime. 
Teachers and schooling systems can recognize and 
support these skills, to identify important attributes of 
collaborative problem solving and digital literacy in student 
work. 
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